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Abstract 

We solve a class of attractor neural network models with a mixture of ID nearest-neighbour 
interactions and infinite-range interactions, which are both of a Hebbian-type form. Our 
solution is based on a combination of mean-field methods, transfer matrices, and ID random- 
field techniques, and is obtained both for Boltzmann-type equilibrium (following sequential 
Glauber dynamics) and Peretto-type equilibrium (following parallel dynamics). Competition 
between the alignment forces mediated via short-range interactions, and those mediated via 
infinite-range ones, is found to generate novel phenomena, such as multiple locally stable 
'pure' states, first-order transitions between recall states, 2-cycles and non-recall states, and 
domain formation leading to extremely long relaxation times. We test our results against 
numerical simulations and simple benchmark cases, and find excellent agreement. 
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1 Introduction 



Solvable models of recurrent neural networks are bound to be simplified representations of 
biological reality. The early statistical mechanical studies of such networks, e.g. [jl], ||, concerned 
mean-field models, whose statics and dynamics are by now well understood, and have obtained 
the status of textbook material ||. The focus in theoretical research has consequently turned 
to new areas, such as solving the dynamics of large recurrent networks close to saturation [Q], 
the analysis of finite size phenomenology ||], solving biologically more realistic models ||, or 
networks with spatial structure |7|, ||, |9|, [H], O]. In this paper we analyse Ising spin models of 
recurrent networks with spatial structure, in which there are two coexistent classes of Hopfield- 
type Q interactions: infinite-range ones (operating between any pair of neurons), and ID short- 
range ones (operating between nearest neighbours only). The study of this type of structure is 
motivated by the interplay between long-range processing via (excitatory) pyramidal neurons 
and short-range processing via (inhibitory) inter-neurons, which is typically observed in cortical 
tissue. In the present model however, and in contrast to early papers on spatially structured 
networks, exact solutions based solely on simple mean-field approaches are ruled out. Due 
to short-range interactions analytical solutions require significantly more complicated methods 
and the present models can be solved exactly only by a combination of mean- and random-field 
techniques (l^, [l3|, O, [l^, [l6| , whereas for the special case in which the system has stored a single 
pattern, a simple (Mattis) transformation allows us to derive the solution via a combination of 
mean-field methods and transfer-matrices. 

Our paper is organised as follows. We first solve the one-pattern case in which pattern- 
variables can be transformed away (thus providing a convenient and exactly solvable benchmark 
case against which to test the general theory) . This also hints at the interesting features induced 
by short- versus long-range competition in the more general model. In particular, already in the 
one-pattern model we find first-order phase transitions, regimes corresponding to multiple locally 
stable states and we find that sequential and parallel dynamics phase diagrams are related by 
simple transformations. We then proceed to the general case, with an arbitrary number of stored 
patterns, away from saturation regimes. For sequential dynamics (Boltzmann equilibrium) we 
adapt the ID random-field techniques as originally developed for site-disordered Ising chains, and 
combine them with mean-field methods; for parallel dynamics (Peretto equilibrium) we adapt 
and combine with mean field methods the procedure in based on 4 x 4 random transfer 
matrices. The disorder- averaged free energy and the order parameters are found, in closed 
and exact form, as integrals over the distribution of a characteristic variable, which represents a 
specific ratio of conditioned partition functions. This distribution is calculated following |jl2| , [D|]. 

In the region where the infinite-range versus short-range competition is prominent, our theory 
predicts a series of (continuous and discontinuous) dynamic phase transitions, and a free energy 
surface with multiple local minima. These features become more prominent, in both number 
and strength, when the number of stored patterns increases, in sharp contrast to Hopfield-type 
infinite-range networks Q, where 'pure state' solutions are independent of the number of patterns 
stored. The transition lines of sequential and parallel dynamics phase diagrams are found to be 
related by reflection symmetries and also parallel dynamics macroscopic equations can describe 
2-cycles rather than fixed-points solutions. Finally we test our theory against several exactly 
solvable cases and against numerical simulations; the latter are found to exhibit interesting but 
extremely slow domain-induced dynamics, with plateau phases. Once equilibration has occurred, 
we obtain excellent agreement between theory and experiment. 
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2 Model Definitions 



We study models with N Ising spin neuron variables er = (a±, . . . , a^) £ { — 1, 1} , which evolve 
in time stochastically on the basis of post-synaptic potentials hi(cr) (or local fields), following 
the Glauber-type rule 

Prob[a i (t + l) = ±l] = i[l±tanh[/3^(o-(t))]] hi(<r) = £ J ijaj + 0* (1) 

The parameters Jy and Oi represent synaptic interactions and firing thresholds. The (non- 
negative) parameter (3 controls the amount of noise, with (3 = and (3 = oo corresponding 
to purely random and purely deterministic response, respectively. If the interaction matrix is 
symmetric, both a random sequential execution and a fully parallel execution of the stochastic 
dynamics (|l|) will evolve to a unique equilibrium state. The corresponding microscopic state 
probabilities can both formally be written in the Boltzmann form Poo(°") ~ ex P[~ PH(cr)], with 
associated Hamiltonians [l7| (since Peretto's pseudo-Hamiltonian Hamiltonian H pax depends 
on /?, the associated statistics are not of the Boltzmann form): 

H seq ((r) = ~ o < T iJij (r 3 ~ ^ ( 2 ) 

ij^j i 

#par(<T) = ~^log2cOSh^( < T)]-^e i (7 i (3) 

In both cases, expectation values of order parameters can be obtained by differentiation of the 
free energy per neuron / = — limAr_* 00 (/3A r )~ 1 logger ex P[ — PH(<j)], which acts as a generating 
function. For the parameters Jy and 9i we now make the following choice n: 

J 1 

Jij = if + Jlj(kj+l + 6i,j-i) 0i = 6 (4) 

fj.=i ^=1 

This corresponds to the result of having stored a set of binary patterns {£ , . . . , £ p } with £^ S 
{ — 1,1} in a one-dimensional chain of neurons, through Hebbian-type learning, but with different 

(2) (2) 

(potentially conflicting) embedding strengths Js and J\ associated with the short-range versus 

the infinite-range interactions. We will choose p -C N. The parameters and control 
uniform contributions to the interactions within their class. Taking derivatives of / with respect 

(2) 

to J) produces expressions involving the familiar 'overlap' order parameters: 

sequential : m 2 = -2 — -- = lim ((—V o-j£J 2 )eq 

QJY N^oo Jy '—r 

parallel : m 2 = --^ = ^hmj(i^ 



1 Competition between a different type of uniform infinite-range and random nearest-neighbour interactions 
has been studied recently in [63, for sequential dynamics only. 



3 



where the brackets (. . .) eq denote equilibrium averages, m = (rn\, . . . , m p ) , and X{ ■ Xj 



^2u x i^- Note that for j] 2 -* = j]p = we obtain the simpler model 



Jij = -A + «/ s (<5i,j+l + = # (5) 

The Mattis transformation o"j — ► cijfj maps this model onto 

Jij = -TrtiZj + Ja(8i,j+1 + 'V,, lUX/ #t = ( 6 ) 

which corresponds to the result of having stored just a single pattern £ = (£i,...,£at) £ 
{—1,1}^. Taking derivatives of / with respect to the parameters and J s in (||) produces 
our order parameters: 

seq : m = -g = Km 1 £ <a;> eq a = - = Km 1 E^a^ 

1 (9 f X 1 (9 ^* 1 

Par : m = - 2 00 = JSo N ^> i)eq ° = " 2 9^ = J&, iV ^> +1 tanh ^( CT )])e q 

where we have simplified the parallel dynamics observables with the identities 

(cr i+ itanh[/?/ij(<x)])cq = (<r<-i tanh[/3/ii(o-)]) eq and (tanh[/?^(<x)]) eq = (ai) eq 

which follow from (|]) and from invariance under the transformation i — > N — i (for all i). For 
model (||) m is the average neuronal activity. For sequential dynamics a describes the average 
equilibrium state covariances of neighbouring neurons, and for parallel dynamics it gives the 
average equilibrium state covariances of neurons at a given time t, and their neighbours at time 
t+1 (the difference between the two meanings of a will be important in the presence of 2-cycles). 
For model (|6|) one similarly finds 

i i 

The observable m is here the familiar overlap order parameter of associative memory models 
EI, 0, which measures the quality of pattern recall in equilibrium. Note that m,a G [—1, 1]. 



3 Solution and Phase Diagrams for p = 1 

Before we proceed to the solution of the general model (|j) we first solve the relatively simple 
situation, where a single pattern has been stored following (|5j) and (||). This has the following 
advantages: it being a simpler version of (|j), it allows us to explore general features and build 
intuition, without as yet any serious technical subtleties, as brought up by the general model. At 
the same time it provides an excellent benchmark test of the general theory to which it should 
reduce for Jg = = 0. For the remainder of this section our analysis will refer to model (0). 
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Solution via Transfer Matrices 

In calculating the asymptotic free energy per neuron / it is advantageous to separate terms 
induced by the long-range synapses from those induced by the short-range ones, via insertion 
of 1 = J dm 5[m — jjJ2i a i\- Upon using the integral representation of the ^-function, we then 
arrive at 

/ = - lim -3- log / dmdm e -^(™.*) 
J N^oo BN J 



with 



seq (m, m) = -imm - md - ]^Ji>m 2 - — log -R seq (m) 



1 

(3N 



-imm — mO — — — log R paT (m, m) 



The quantities R contain all complexities due to the short-range interactions in the model. They 
correspond to 

ae{-i,i} N 

R paI (m,m) = e- 1 ^^ ]J log[2 cosh[0J e m + 89 + (3J s (a i+1 + <*_{)]] 

crei-i.i}^ i 

They can be calculated using the transfer-matrix method giving 



e (3J s -iflm e~@ Js 



S «1 J sec l I e -P.)s e f3J s +i/3m 



N _ / 2cosh[/3u7+]e-^ m 2cosh[3w ] \ 

R p3X {m,m) - Tr [T par j T par - ^ 2cQsh[0WQ] 2 cosh[/^_] J 

where wq = Jim + 9 and w± = wq ± 2J S . The identity Tr [T N ] = A+ + A^, in which A± are the 
eigenvalues of the 2x2 matrix T enables us to take the limit N — * oo in our equations. The 
integral over (m, m) is then for N — ► oo evaluated by steepest descent, and is dominated by the 
saddle points of the exponent eft. We thus arrive at the transparent result 

{4> secL (m, m) = —imm — m9 — ^J^m 2 — 4 log A+ q 
1 par (7) 

par (m, m) = —imm — mO — -m log A+ 

where A+ q and A+ ar are the largest eigenvalues of T seq and T par : 



A seq = e PJs cos h[i/3m] + e 2/3Js cosh 2 [i/3m] - 2sinh[2/?J s ] 2 

Af r = cosh[/3u; + ]e- i/3A + cosh[3w.]e lf3lh + [cosh 2 [^+] e - 2/3 ™+ 

+ cosh 2 [/3w_]e 2/3i ™ - 2cosh[/3w+] cosh[/3u;__] + 4 cosh 2 [/?w ] 



For simplicity, we will restrict ourselves to the case where 9 = 0; generalisation of what follows 
to the case of arbitrary 9, by using the full form of (^), is not significantly more difficult. The 



expressions denning the value(s) of the order parameter to can now be obtained from the saddle 
point equations d m cp(m, rh) = c^</>(m, to) = 0. This is a straightforward differentiation task for 
the sequential case. For parallel, one obtains a set of coupled non-linear equations, namely 

to = J-(m, rh) rh = J-(fh, to) (8) 

where to = — irhj Jg and , •) corresponds to 

T{p, q) = A(p, q)- 1 (e 2/3J = sinh[/3^(p + q)) - e~ 2 ^ smh[(3J e (p - <?)]) 



A(p,q) = [e 2/3Js sinh 2 [/3J £ (p + q)] + e~ 2l3Js sinh 2 [/5J £ (p - q)] + 2 cosh 2 [/3J^] + 2 cosh 2 [(3J e q] 

with A(p, q) = A(q,p). We will now show that the parallel dynamics fixed point problem (||) 
admits the unique solution to = sgn[J^] to, by the following argument: 

For Jg > : to = to since < (to — to) 2 = Q(m, to) (to — to) sinh[/?J^(TO — to)] < 
For < : to = —to since < (to + to) 2 = J7(m, to) (to + to) sinh[/3J£(m + to)] < 

where Q(m,rh) = 2 e~ 2/3Js A(to, to) -1 > 

Insertion of these solutions to the original function 7 s (■ , •) allows us to reduce (||) to a simple 
ID fixed-point problem, similar in structure with what follows from the sequential case, namely 

sequential : to = imJg, m = G(m; Jg, J s ) 

parallel : rh = imJg, to = G(m; Jg, J s ) for Jg > (^) 

to = —imJg, to = G(m; — J^, — J s ) for < 



with 



, , smhlfl Jgm] , 

G(to; J/, J s ) = 1 J (10) 

smh 2 [P J £ m] + e - 4 ^ 



The macroscopic observable a is generated by differentiating the reduced free energy per neuron 
sequential : to = imJg, a = F(m; Jg, J s ) 

parallel : rh = imJg, a = F(m; Jg, J s ) for Jg > (H) 

rh = —imJg, a = F(m; —Jg, — J s ) f° r Je < 

with 



cosh[P J e m] J sinh 2 [P J e m] + e" 4 ^ + sinh 2 [/?J £ to] - e ~ 4/3Js 

F(m; Jg, J s ) = , (12) 

cosh[/3J£TO]Wsinh 2 [/?j£TO] + e" 4 ^ + sinh 2 [/? J £ to] + e" 4 ^ 



Note that in the absence of short-range interactions we recover the familiar Curie- Weiss law 
to = tanh [/JJ^m] whereas in the absence of long-range interactions (g) and (|ll|) reduce to to = 
and a = tanh[/3J s ], as they should. Finally, it is worth noting that equations (p|,[l0|,ll,12) allow 
us to derive the physical properties of the parallel dynamics model from those of the sequential 
dynamics model via simple parameter transformations. 
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Phase Transitions & Phase Diagrams 



Our main order parameter m is to be solved from an equation of the form m = G(m), in 
which G{m) = G(m; Jg, J s ) for both sequential and parallel dynamics with Ji > 0, whereas 
G(m) = G(m; —Je, —J s ) for parallel dynamics with Ji < 0. Note that, due to G(0; Jp, J s ) = 0, the 
trivial solution m = always exists. In order to obtain a phase diagram we have to perform a 
bifurcation analysis of the equations (JQ^pLCl) , and determine the combinations of parameter values 
for which specific non-zero solutions are created or annihilated (the transition lines). Bifurcations 
of non-zero solutions occur when simultaneously m = G(m) (saddle-point requirement) and 
1 = d m G(m) {m is in the process of being created/annihilated) . Analytical expressions for the 
lines in the (PJ s ,PJi) plane where second-order transitions occur between recall states (where 
m ^ 0) and non-recall states (where m = 0) are obtained by solving the coupled equations 
m = G{m) and 1 = d m G(m) for m = 0. This gives: 

sequential : PJi = e _2/3Js 
cont. trans. : „ T (13) 

parallel : pj e = e~ 2 ^ and pj e = -e 2 ^ 

whereas for the macroscopic observable a we obtain 

sequential/parallel : a = tanh[/3J s ] 



If along the lines ( fL3| ) we inspect the behaviour of G{m) close to m = we can anticipate 
the possible existence of first-order transitions, using the properties of G(m) for m — > ±oo, 
in combination with G(—m) = —G{m). Precisely at the lines (p^) we have G{m) = m + 
iG'"(0).m 3 + C>(m 5 ). Since limm^oo G(m) = 1 one knows that, when G"'(0) >0, a discontinuous 
transition must have already taken place earlier, and that away from the lines ([0|) there will 
consequently be regions where one finds five solutions of m = G(m) (two positive ones, two 
negative ones and m = 0). Along the lines ( |l~3|) the condition G"'(0) > translates into 



sequential : (3Je > and (3J S < — |log3 ^ 
parallel : \@Jt\ > V% and \(3J S \ < — 2 log 3 

In the present models it turns out that one can also find an analytical expression for the discon- 
tinuous transition lines in the ((3J s ,f3Je) plane, in the form of a parametrisation. For sequential 
dynamics one finds a single line, parametrised by x = (3 Jim € [0, 00): 



/ x 3 1 

discord, trans. : PJi(x) = \ , (3J s (x) = —-log 

y x — tanh(x) 4 



tanh(x) sinh 2 (x) 
x — tanh(x) 



(15) 



This can be verified by explicit substitution into @. Since this parametrisation (|l5|) obeys 
PJ S (0) = — j log 3 and /3J^(0) = \/3, the discontinuous transition indeed starts precisely at the 
point predicted by the convexity of G(m) at m = 0, see (14). In the limit x — * 00 the slope of 
(|l5| ) approaches f3 Ji(oo)/ f3 J s (00) — > —2. For sequential dynamics the line (15) gives all non-zero 
solutions of the bifurcation requirements m = G(m) and 1 = d m G{m). For parallel dynamics 
one finds, in addition to (|l5|), a second 'mirror' transition line, generated by the transformation 
{/3J t ,pj 8 }»{-pj t ,-0J a }. 

Having determined the transition lines in parameter space, we can turn to the phase dia- 
grams. Figure [l] shows the phase diagram for the two types of dynamics, in the (PJ S , PJe) plane 
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Figure 1: Left: Phase diagram for sequential dynamics, involving: (i) a region with m — only (here 
a = tanh[/3 J,]), (ii) a region where both the m = state and two m ^ states are locally stable, and 
(Hi) a region with two locally stable m ^ states (with opposite sign, and with identical a > 0). All 
solid lines indicate second-order transitions, whereas the dashed lines indicate first-order ones. Right: 
phase diagram for parallel dynamics, involving the above regions and transitions, as well as a second set 
of transition lines (in the region Jg < 0) which are exact reflections in the origin of the first set. Here, 
however, the two m ^ physical solutions describe 2-cycles rather than fixed-points, and the Ji < 
region describes simultaneous local stability of the m = fixed-point and 2-cycles. 

(note: of the three parameters {(3, J s , Jg} one is redundant). For sequential dynamics we find 
(i) a region where m = only (ii) a region where the trivial as well as two non-trivial states (a 
positive and a negative one) are all locally stable (selection will thus be based on initial condi- 
tions) and (Hi) a region where only states are localy stable. The transitions (i) — > (Hi) 
and (ii) — ► (Hi) are second-order ones (solid lines of figure |l|) whereas the transition (i) — > (ii) 
is first-order (dashed line). In region (i), where m = only, one finds a = tanh[/3J s ] whereas in 
regions (ii) and (Hi) a is given by the full expression of (|Tl|). For parallel dynamics we find, in 
addition to the sequential phase transitions, a second set of 'mirror' transition lines generated 
by {{3Ji,f3J s } i ^ {— {3Je, —f3J s }. In contrast to the sequential case however, here in the region 
j3 Ji < and (3 J s < where m/0 can be a physical state (lower left corner of the phase diagram 
of figure [T]) one finds 2-cycles between the two m/0 (positive and negative) recall states. This 
can be inferred from the exact dynamical solution that is available along the line J s = (see 
e.g. ||), given by the deterministic map m(t + 1) = tanh[/3 Jgm(t)\. This map gives a stable 
period-2 oscillation for f3Jg < —1, of the form m(t) = ( — l)*m*, where m* = tanh[/3| Ji\m*]. In 
the 2-cycle region one has a = lini7v- +0 o jj Y^A^i+i tanh[/3/ij(<r)]) < 0. This can be understood 
on the basis of the (parallel dynamics) identity (o"j + i tanh[/3/ij(er)]) = (ai + i(t)ai(t + 1)). 

We find that in contrast to models with nearest neighbour interactions only ( Jg = 0, where 
no pattern recall will occur), and to models with mean-field interactions only (J s = 0, where 
pattern recall can occur), the combination of the two interaction types leads to qualitatively new 
modes of operation, especially in the competition region, where Jg > and J s < (Hebbian 
long-range synapses, combined with anti-Hebbian short range ones). The novel features of the 
diagram can play a useful role: the existence of multiple locally stable states ensures that only 
sufficiently strong recall cues will evoke pattern recognition; the discontinuity of the transition 
subsequently ensures that in the latter case the recall will be of a substantial quality. In the 
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J* M 

Figure 2: Left: alternative presentation of the competition region of the sequential dynamics phase 
diagram of figure [l]. Here the system states and transitions are drawn in the (Je,T) plane (T = /3 _1 ), 
for J s = —1. Right: the magnitude of the 'jump' of the overlap m along the first-order transition line 
(dashed lines in figure [l]), as a function of /3Jg(x), x S [0, oo), see equation (ff5[). 



case of parallel, similar statements can be made in the opposite region of synaptic competition, 
but now involving 2-cycles. Since figure [l| cannot show the zero noise region (j3 = T _1 = oo), we 
have also drawn the interesting competition region of the sequential dynamics phase diagram 
in the (Je,T) plane, for J s = — 1 (see figure [2|, left picture). At T = one finds coexistence of 
recall states (m ^ 0) and non-recall states (m = 0) for any Ji > 0, as soon as J s < 0. In the 
same figure (right picture) we show the magnitude of the discontinuity in the order parameter 
m along the discontinuous transition line, as a function of (5J^. 

Finally we show, by way of further illustration of the coexistence mechanism, the value 
of reduced exponent (j) S eqijn) given in (|^), evaluated upon elimination of the auxiliary order 
parameter rh: 4>{m) = 4> seq (m, imJi) . The result, for the parameter choice Jg) = (2,3) and for 
three different short-range coupling stengths (corresponding to the three phase regimes: non- 
zero recall, coexistence and zero recall) is given in figure |||. In the same figure we also give the 
sequential dynamics bifurcation diagram displaying the value(s) of the overlap m as a function 
of (3Ji and for (3J S = — 0.6 (a line crossing all three phase regimes in figure [l]). 



4 Solution and Phase Diagrams for Arbitrary p 

In the general case where p > 1, the pattern variables {£f } can no longer be transformed 
away as in model (||); here in order to arrive at expressions for the order parameters one has to 
perform the disorder average of the free energy over the realisation of {q }• Due to the addition 
of nearest neigbour interactions, however, the disorder average has become significantly more 
complicated than that in infinite-range models, even in the loading regime away from saturation 
(i.e. limN^cop/N = 0). We perform the disorder average via a suitable adaptation of ID 
random-field techniques, see e.g. [12, 13, p]]. These are based on the derivation of a stochastic 



process in which observables of systems of size N are mapped to observables of systems of size 
N+l. We will assume that in the thermodynamic limit this process leads to a unique stationary 
state. The free energy is then given as an integral over the distribution of a characteristic ratio 
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Figure 3: Left: Free energy per neuron <f>[m) = 4> scq (m,imJ£) as derived from equation (Q) for ((3, J£) = 
(2,3). The three lines correspond to regimes where (i) m ^ only (dashed line: J s — —0.2) (ii) trivial 
and non-trivial states are both locally stable (solid line: J s = —0.8), and (Hi) m = only (dot-dashed 
line: J s = —1.2). Right: Sequential dynamics bifurcation diagram displaying for (3J S — —0.6 the possible 
recall solutions. For small j3Ji 7 m — is the only stable state. At a critical fiJi given by (|l5|), m jumps 
to non-zero values. For increasing (3Jt the unstable m ^ solutions (thin lines) converge towards the 
trivial one until (3Ji — exp(—/3J s ) (here: j3Ji ~ 3.32) where a second-order transition takes place and 
m = becomes unstable. 



of conditioned partition functions, and the order parameters follow via differentiation. 
Adaptation of RFIM Techniques 

We introduce the notation m* = (mo,m), m* = (fho,rh), = where £° = 1. We 

separate the overlap order parameter via insertion of 1 = J dm* 5[m* — 1/-/V^<jj£*], and we 
replace the J-functions by their integral representation giving 



/ = _ i im J_ bg / dm*dm* e -^Mm*,m*) 



where 



(m*,m*) = -im*. m *-m e-^4 l) m 2 -^4 2) ™ 2 - ^ logi?^ q (m*) (16) 



1 



(^ ar (m*, m*) = -im* • m* - m 9 - — log R?x(m*, rh*) (17) 

The quantities i?/v(m*, rh*) contain, as in model (||) in section ||, the summation over er = 
(o"i, . . . , (Tjv) and the short-range neuron interactions 

^ q (m)= Yl ^q(fr) R p N ar (m*,fh*) = £ .f par (<x), (18) 

^ scq (cr) = e -^E^™-€ ^Ei^i+i^+i 

f P M = e-^^^^'C TJlog^cosh^Uf + ^J^ + ^+xJ^)]] 

i 
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where the short-hands Jfi + i and jf correspond to: 



j = e 



rn 



Each of the quantities in fllq ) is now separated into different constituent parts, defined by 
conditioning on either the state of the last neuron in the chain (sequential dynamics), or on the 
states of the last two neurons in the chain (parallel dynamics): 



pseq 
pseq 



scqV 



ppar 

- rt JV,TT 

ppar 
ppar 
ppar 



El 



par \ 



<W -i 

"o-jv_i,1 0(7 JV, — 1 



(19) 



J 0"JV-1; 



J 0"JV-1; 



-1 <W,1 
-1 °"(TJV,-1 



(20) 



We now add an extra neuron to the chain. After some simple bookkeeping and assuming non- 
periodic boundary conditions we derive the following recurrence relations: 



seq 




R 



N,N+1 



■N,1 



par : 



Rn+i,ii 
Rn+i,h 



— 2Q~N,N+l 



( e -*f>™-*-t*N+i AN) e -^'4 +1 AN) \ 
rF (+,+) n- 

i 



~cF — 



\ 



(+.-) 



3^" 

*N-1,N 



N-1,N 



(21) 




2Q 



N.N+l 



e ifnX+i .(at) 



rF 

'N-1,N 



iprh* -Cn+i ,(N) 



\ 



(+,-) 



JV+i .(AT) 




with the short-hand notation: 



A 



A(± ±)(£ n -l ' %n ' %n ' €n+l > £n " m ) — cosn [/^(^n ± <^n-l,n ± J»,n+l)] 



(23) 



Qn.n+l = Q ± i€n ' £n+l J £n ' m ) = cos h[/?(^n+l ± ^n,n+l)] 

The fact that parallel dynamics leads to a recurrence process given by 4 x 4 random matrices, 
instead of the simpler 2x2 random matrices of the sequential case, is due to the appearance 
of short-range couplings of the form cosh[(3(J- + ai-iJf_ li + <7j + iJ/ i+1 )] in Jig) , rather than 
the more familiar exponentials. This, in turn, is an immediate consequence of the form of the 
pseudo-Hamiltonian (||). We also note that in the parallel case the two reduced 2x2 matrices 



differ only by the prefactors n+i- ^he recurrence matrices (^TJ- 22 ) will form the basis for 
evaluating the free energy per neuron, which follows from 



pscq pscq 
7V,1" 11 N, I 



p par 
N 



ppar R par R par p par 

TV, IT N^l ' N,i1 ' N II 



(24) 
(25) 
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The Stochastic Process for Conditioned Partition Functions 



sequential 
parallel : 



k 



n+l 



k {1) ~< 



(26) 



In the spirit of [12, 13] we next define the ratios between the conditioned quantities (|19|) and 
(|20|), and study their stochastic 'evolution' generated by adding new neurons successively to the 
chain: 

-2f3im -Cn+i Rn +^i 
^n+l,T 

-2/3im*-£I +1 R n+l,U 
Qn,n+1 Rn+l,U 



k {2) -e 

Si+1 — e 



-2/3im*-£* +1 -flw+i,U 
Rn+i,l\ 



k 



(3) 
n+l 



Qn.n+1 R n+l,U 



(27) 



Due to their dependence on the random variables {£}, each of these quantities is a stochastic 
variable. Insertion of (^-22) into the above relations gives 



sequential 



parallel 



"n+l 



v n+l 



u(3) 
v n+l 



,+1 + k n e 2/3ilfl *-C e P J n,n+l 
\ run ~r -fi/ \ n>n run 



(28) 



v n+l 



A 



(n) 

(+.-) 



+ 4 



(n) 
(-.-) 



.(3) 



«) fr (2) , ,(«) fr (3) 

_|_ run ~T~ -^l/ ^ run run 

n) t (l) t (2) , 4 (n) , (1) , (2) , (3) 

^ rZ n + A ( _ _ } ft n K n K n ^^.^ 



An) An) , (3) 

^(+,+) + Kn 



4(«) J2) , ,(n) 
-^(-l ^ n^n T -*^-/ \ run ru n 



1,(3) 



(29) 



For parallel dynamics we observe that, if the identity k n ^ = kffl is true, then also k^li = 
Furthermore, it can be easily checked that this is the case for n = 2. We are thus guaranteed that 



run 



(1) 



k n 2 ^ for all n > 2, and now all three quantities {k\ x> , k\ z> , ktf 1 } can (for any i = 2, . . . , N) 



(i) 1.(2) fc (3)- 



(i) 



be expressed in terms of a single stochastic variable which we will take to be k, 
simplifies considerably the description of the parallel dynamics stochastic process: 



kj . This 



k 



(i) 

n+l 



k 



n+l 



1.(2) 
"n+l 



^(+,-) + ^(--) K-ie 2 ? 1 ™^ 



A 



(n) 

(+,+) 



+ A 



(n) , 2^im •£„_! 

(— ,+) n ~l 



2,(3) _ > 2/?im*-£* 
"'n+l ~~ e 



We have thus obtained ID stochastic maps for both sequential and parallel dynamics: 
seq : k i+1 = Vseq (kj ; ij • t/+i , rh* ■ Cj I V j < 



n,n + l + fc„ e ^™-i 



par : 



cosh[/3(jf + J?,, - J? i+1 )] + e 2 ^™*^*-^ cosh[/3(jf - J?,, - J: 



(30) 



(31) 



i,i+l) 



cosh[/3(j/ + Jt M + J M+i)l + e 2 ^*^-^! cosh[/3(Jf - J?_ M + J? i+1 )] 

We observe that, in contrast to the sequential dynamics mapping, parallel dynamics distinguishes 
between even and odd sites, and produces two independent sets of stochastic variables {kj}. 
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Disorder Averaging and the Free Energy 



One can now express the non-trivial part of the free energy in terms of the above ratio's ki, 
using (HH): 

Upon also using (2^-27) and p^ - p9[ ) iteratively, in order to map the quantities {R^, -^Tt} 
onto {R^-i tj-^n-i tT J" ^ or a ^ n — one can wr ^e the above expressions in the form: 



log R^(m*) = - J- ^ log [e? J w + e 2 ^™*"^ 



/3iV 
1 



1 



— logi^ TT (m*,Th*) = -^E lQ g 2cosh[/3(Jti + + J4i, i+2 )]+ 



/3iV 



+2cosh[/3(4 fl - J* i+1 + Jf+i )i+2 )] e 2 *™*^ kj 



i ' i 

where the terms Q^. are defined at d23|). For N ^ oo the above expressions are self-averaging 
and the boundary terms vanish, giving 

1 



sequential : 



-E dk' P s (k',t',£) log 



(32) 



parallel : 



i(m*-r') r Jlog2-^(lo 



/9 



(33) 



in which the probability distributions P s (. . .) and P p (. . .) are defined as follows: 



sequential: P s (k',£',£) 



1 



i 



1 

2p 



(34) 



1 



parallel: P p (fc", = J im E < S W ~ k ^ ~ 6-d ~ &] ^ ~ 



N^oo N 

i 



cq 



(35) 
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In order to arrive at factorisation of these joint probability distributions we have used the fact 
that, due to the form of the stochastic maps ( p0[ ) and (|3l|), the quantities {k n } are (for both 
types of dynamics) independent of {£ n+1 }, i.e. of the pattern variables at the next site. The 
dependence on however, does not allow us to write down immediately ( |30| - |3T| ) in terms of a 

Markov process. We thus introduce the auxiliary variables {A n }, so that {£ n } itself becomes one 
of the stochastically evolving variables, and the extended process becomes indeed Markovian: 



seq : 



k 
X 



k 



i+l 



par : 
Equivalently: 

sequential : 

parallel 



1 i+l 



par 



seq 
Jfe> 

A , 



/ i 
i €i+li€ 



i-1 



/ ip seq (ki; Xi ■ € i+1 ,rh* ■ A*) \ 



W^) = ^E Jdk'w s 



k' 
X' 



k 
X 



,X') 



where for the transition probabilities W s [. . .] and W p 
sequential : W s 



we define: 



k 
X 



Pf{k",X") 



(36) 
(37) 



k' 
X' 



k 
X 



parallel : W p 



k" 
X" 



k 
X 



= (5 tx 5[k - ^ scq (k',X' ■ trh* ■ X*')] }£ 
( S tX 5 l k ~ ^M*"> X " ■ £'> Z'-Z,™- Z', ™* • A*")] 



The evaluation of the non-trivial part of the free energies (|32|j33| ) has now been reduced to 
determining the (stationary) distributions of (36-|37|), which will be denoted as P^(fe) and P^ Q (k). 
To achieve this final objective we follow [|l^, 13] and introduce the integrated densities P(k) = 
/„* dzP(z). After some algebra to eliminate the ^-functions of the transition probabilities via the 
identity Jdx 5[g(x)]f(x) = f (g mv (0)) /\d x g(0)\, we then obtain the following recursive relation: 



sequential: Pf +1 (fc,A) = 1 £ if (B s (k ; A' • A , m* • A*'), A' 

A' 



parallel: P? +1 (k,\) 



(38) 



£ £f? (S p (fc ; A" - , • A , m , m* ■ A*'), A") (39) 
A" £' 



where the functions B s (k) and B p (k) correspond to 



-/3J S 



e /3J s — g ^ s 



-2f3irh*-X* 



B p (k) 



kA 



(+,+) 



.4 



i4 



(-,-) 



(+.-) e -2/3ilfl*-X* 



(-,+) 



For notation simplicity we have removed the arguments of J s and A^^). These correspond to 

J s = UX'-X) A (±i±) = A (±>±) (A" A, m-SO 

= J s (1) + J s (2) A' • A = cosh[/3(J<(m • m , 0) ± J S (A" • £') ± J s (£' • A))] 
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We have thus derived as yet fully exact expressions for the disorder-averaged free energies (32 ,33), 
as integrals over the distribution of the stochastic quantities {k seq , k pa-T }, which can be evaluated 



by numerical iteration of the recursive relation (pq-39). 



The 'Pure State' Ansatz and Symmetries of the Model 

We will now look for the so-called 'pure-state' solutions: we will assume that for N — ► oo 
there is only one component m^ of order 0(1) (for simplicity we will take [i = 1) whereas for 
all /j, > 1 m M is of order 0{N^^): m* = (mo, m, 0, . . . , 0) and rh* = (mo,m, 0, . . . , 0). This 
is the standard ansatz made in (infinite-range) associative memory models, which gives the 
dominant states of the system. The fact that this is also true for the present type of models 
is supported by numerical simulations, see section ^. Using this simplification we will now 
prove that the stationary integrated densities -P^(. . .) and P^i- ■ .) are independent of all non- 
condensed pattern components. This will be shown by induction. We will show that if Pf{k, A) 
and Pi(k, A) are independent of A M for some fi > 1, then Pf +1 (k, A) and P[ +1 (k, A) will also be 
independent of A^. This will then immediately imply that if the densities are independent of 
{A^} for all [i > 1 at step % of the process ( |38|]3"9| ) , this will remain so for all \i > 1 at any step 
j > i. Usage of the spin-flip operator F^x = (x%, . . . , — x M , . . . , x p ) allows us to write 

seq : Af +1 = Pi +1 (k,F^X) - Pf +1 (k,X) 

= w H\' {Pi {B B (k; X' • A, m + mA'), F^X') - F? {B s (k; X' • A, m + mA'), A') 

par : A? +1 = P? +1 (k,F,X) - P^ +1 (k,X) 

= ^ E\"£ [P? (Bp(k; A" • €' • A, m + m£', m + mA"), F^X 1 ) - 

Pf (B p (k; X" ■ £' • A, m + m£', m + mA"), A' 

If at step % the identities A| = A? = are true, then P i {B{k),F ill X) = Pi(B(k),X) (for both 
types of dynamics) and thus Af , 1 = A? +1 = 0. Upon choosing suitable initial conditions we can 
thus construct equilibrium integrated densities with the stated property; combination with the 
assumed ergodicity of the process then implies that the unique solution must have the property. 
This completes the proof. We can consequently write the stationary integrated densities in 
the form: P^ Q (k,X) and P^ Q (k,X), where A € {—1,1} corresponds to the condensed pattern 
component: 

sequential: P! +1 (k,X) = \ Yl Pf (B s {k ; X' X , m + mX'), X') (40) 

2 A'=±l 

parallel : Pf +l {k, X) = - if (B p (k ; A"f , £'A , m + m£' , m + mA"), A") (41) 

\",£'=±l 

One can also exploit the fact that the pattern variables {A} aeq and {A} par appear in inner- 
products only, to simplify the non-trivial integrated expressions of equation ( p2]j3^ ). Since the 
argumentation will be qualitative we will forget about the details of these expressions and we 
will denote the integrated logarithmic expressions simply by 



* S (A' • A) = log 



e /3J s (X'-X) + e -/3J 3 (X'-X) + k l e 2f3i(rh +rh\') 
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<f p (A"- A) = log 



A, +t+) (A" • ■ A, C'm) + (A" • • A, £'m) *" e ^+-A») 



Upon using the gauge transformations: A M = t^A^ for sequential dynamics and X" u = 



A^ = V^'fi f° r parallel dynamics, where and are auxiliary Ising variables, the non-trivial 
parts of expressions ( |32| ) and ( |33| ) -corresponding to the integrals over the stochastic variables 
{fej} after the distribution factorisation (34) and (35) and the 'pure state' ansatz- take the form: 

sequential : — ^ ^ J dk' P^(k' , A') $ S (A' • A ; k', m + mX') 



A' A 

p — 1 
TP 



A' T 



p — 1 
2P 



E E K 



A'=±1A/V=1 



(fc 7 , A') $ S (2M -p; k',m + mX') 



parallel : 



i E E E / ^" A ") $ p( A " • £'> • A > < ; fc "> + mA") 

A" £' A 



P-l 

2 2p 



i- a' v 

{p-if 



E E E / ^" p £( fc "> nC) $ P (n + E v E ^> ; ™o + mn£') 



p p 



2 2p 



E E E E 

ri=±l^'=±lA^=2A^ = l 



dk" 



p — 1 
J\f T 



M 



x $ p (ri + 27V T - p + 1, 2M, - p, m£' ; fe", m + mr^') 



where M- an d Mi represent the number of neurons with states equal to +1 in the configurations 
of r, 77 respectively. We have thus replaced all summations over the 2 P configurations of the 
vectors {A, A'} seq and {A", £ , A} par by summations over binary- and p-state variables. 

Phase Diagrams 

In order to calculate phase transitions and draw phase diagrams we will first calculate the free 
energy surfaces (16|-17|), which at this stage are still functions of the order parameters m and rh. 
For simplicity we will now set 9 = 0. The distributions P^O • •) an d P£oi- ■ •) can be calculated 
numerically via iteration of ( f5|]4l| ), and bifurcations of the non-trivial values for the pure state 
overlap from the trivial solution (if they exist) will then be given as the solutions of the following 
fixed point problems: 



sequential : m = dimF seq (irh) at im = —J\ m 

parallel: m = dimF pSLT (irh,rn) at irh = d m F par (im,m) 



(42) 



with 



F seq (im) 



p-l 



2p v i&xfcx 



E E dk' 



p 

Mr 



PL(k',X') x 



x log 



J3[4 1) +4 2) (2Af T -p)} + e -f3[ji 1) +4 2) (2Mr-p)} £' e 2/3iAA' 



(43) 
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Figure 4: Phase diagrams of model (Q) for p = 2 and with jj 1 ^ = jj; 1 ' — 0, for sequential (left picture) 
and parallel dynamics (right picture). Lines separate regions with different numbers of locally stable 
solutions for the pure state overlap to, calculated from equations ( p^||4~4| ). In region A the only stable 
solution is the trivial one. The transition A^B is second-order, whereas B^C and F^C are first-order 
(see also figure ||). In regions B,C and F there are 1,2 and 3 locally stable to > states, respectively 
(see also free energy graphs, figure 0). 



F pSLl (im,m) 



2 2 P(3 



E E E E 

ti=±1£'=±1JV t =2JV»,=1 




x lo 



2 cosh[/3 jf W + W4 l) + 04 2) ( r i " 2 P + ^ + 2M V + 1)] + 
+2 cosh^ + /5 J i 2) [2M, - 27V r + n + 1]] fe" e 2/3iArif 'l 



p — 1 

" 2 2 p/3 



E E 

f=±ljV=l 




cosh[/3 jf } m£ + J s w + J] Zj (2AA - p)] 
cosh[/? J £ (2) m£ - J s (1) - J S (2) (2A/ - p)] 



(i) 



r( 2 ), 



(44) 



As in the solution of model (||) in section |!| the parallel dynamics fixed-point problem (|42^]44| ) 
takes the form of a set of coupled equations, which makes the evaluation of bifurcation points 
essentially more laborious. 



The solutions of the above equations, for J. 



(i) 



Jp 1 ^ = and for p = 2, are shown in the 



phase diagrams of figure |||. One distinguishes between four different regions, dependent on the 
number of locally stable 'pure state' solutions: region A with m = only, region B with one 
locally stable m > state (and one m < 0), regions C with two locally stable m > states 
(and two m < ones) and region F with three m > states (and three m < ones). Note 
that region F is created at the point where regions C start overlapping. The transition A^B 
is second-order, whereas B^C and F^C are first-order. The two qualitatively different types 
of bifurcations are also shown in figure |5| (left picture), where we draw the solution(s) of the 
overlap m as a function of (3Jt along the line (3J S = —1.8 (a line crossing regions A,B and C). 
The zero noise region (T = /3 _1 = 0) for the phase diagram of figure || is shown in figure || 
(right picture) where we draw the transition lines separating recall regimes A ,B, C and F in 
the (4 2) ,T) plane for J s (2) = -4. 

In equations (43) and (fM|) we observe that, due to the explicit appearance of the variable p in 
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3 4 5 6 7 8 9 10 20 30 



/3jf» Jf» 

Figure 5: Left: Sequential dynamics bifurcation diagram corresponding to the phase diagram of figure 
along the line /3ji 2 ^ — —1.8. The transition A^B in figure His shown here as a continuous bifurcation 
of the trivial solution whereas the two other bifurcations (at (5J\ w 6.76 and f3J]s « 7.7) correspond to 
the first-order transitions B— >C. Right: Alternative presentation of the phase diagram of figure |] drawn 
in the (jf \T) plane (with T = /3" 1 ) for J s (2) = -4. 



the solution of the overlap order parameter (which is due to short-range interactions, originating 
from expressions of the form exp[/3(ji 1 ^ + j] 2 ^ Y^L=\ £f£i+i)L see e -S- (!§)), it will no longer be 
true that the pure state ansatz leads to solutions which are independent of the number of stored 
patterns, as is the case for standard mean-field Hopfield networks. This is also shown in the 
phase diagrams of figure [f| which have been constructed from (^,44) with = J# = 



and for p = 15. We observe a significant increase in the number of transition lines, as well as 

(2) (2) (2) (2) 

additional transition lines appearing in the quadrant J\ , J s > (and also in J £ , Jl < 
for the parallel case). Such effects become more and more prominent as the number of patterns 
increases. It is also worth noting that figures f§ and pimply that all 'pairs' of first-order transition 

(2) (2) 

lines point to the origin of the { J s , J\ } plane. 

Finally, due to the occurrence of imaginary saddle-points in ( |42| ) and our strategy to eliminate 
the variable m by using the equation d m <f>(m, rh) = 0, it need not be true that the saddle-point 
with the lowest value of 4>(m, rh) is the minimum of (complex conjugation can induce curvature 
sign changes, and in addition the minimum could occur at boundaries or as special limits). To 
remove this uncertainty we have evaluated the sequential dynamics free energy 4> seq (m,rh) (|l6| ) 
after elimination of the conjugate variable rh by using d m (f)(m, rh) = (middle row of figure [/]) 
and dfo^im, rh) = (lower row of figure 0) which shows that, although the convexity of the free 
energy graphs is indeed affected, the location of the minima is not. 



5 Benchmark Tests 

We now compare our results with simple benchmark cases. First, the solution of model (|j) 
should reduce to the model of Amit et al Q, for regimes where p <C N, upon removing short- 
range connectivity, i.e. for jj 1 ^ = = 0. Indeed we find that in this limit the probability 
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Figure 6: Phase diagrams of model (^) for p = 15 and with Jg — Jg — 0, for sequential dynamics (left 
picture) and parallel dynamics (right picture) . Lines separate regions with different numbers of locally 
stable solutions of the pure state overlap m, calculated from equations ( [43f|44| ). Compared with the cases 
P = 1 (figure |l]) and p — 2 (figure ^]), we observe a significant increase in the number of transition lines, 
caused by the explicit dependence of equations (p3|-p4|) on p. The diagrams involve three regions: region 
A where m = only, region B with one m > locally stable state and regions C (appearing inside each 
of the transition-line pairs) with two locally stable m > states. The transition A^B is second-order, 
whereas all transitions B— »C are first-order. Also note the appearance of further transition lines in the 

(2) (2) 

upper right quadrant, where J) , J s > (and the lower left quadrant for the parallel dynamics case). 



distributions P^ q (k) and P^ T (k) of the stochastic variables ( |30| ) and (31) both reduce to the 
delta peak: 6[k — 1]. This simplifies the solution of our problem and allows us to write for the 
free energies: 



4> seq (m,m) 
4>pa.v(m,m) 



-imomo — imm — m^t 



-imQrriQ — imm — m$t 



-j £ m + -j £ m 



1 



(log 2 cosh[/3z(mo + m£)])£ 



r(2)„ 



■-(log 2 cosh[/3i(m + Th£)\) 6 - -(log 2 cosh[/3(# + jf } m + J^mO])^ 



Simple differentiation with respect to {mo, mo, m, m} verifies that in the mean-field limit the 

pure order parameter solutions reduce to m = (£ tanh[/3(0 + jf^m® + m)])^ and mo = 

(tanh[/J(# + J^mo + m)])^ as they should. 

Our second benchmark test is provided by the exact solution of model (||) , section [| We can 

immediately map model (||) to model (|5|) by setting j]p = jj 2 ^ = 0. We then find that the key 
variables {kj} of equations (2^-29) are given by a simple deterministic map. In fact, for both 
sequential and parallel dynamics we find that k seq and k paT evolve towards the same fixed point: 



sinh[/3im] + y sinh [flim] + e 4:/3Js 



which (at the relevant saddle points) can be verified to lead to the fixed-point equation (||) of 
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model (H), namely 

m = G(m; Ji, J s ) 



with 



G(m; Je,J s ) 



sinh [(3 Jim] 



sinh 2 [pJem] + Js 



Thirdly, we have also compared the free energies of model @), as given by our present 
solution, to that which one finds when using the alternative random-field technique of [15]. The 
latter relies on performing the spin summations in R = J2a •^ r (°") (HH ) an d deriving appropriate 
functions A(£ ■ £') and B(£ ■ £') such that the identity cosh[/3(J s (£ • £')o J - im£)] = exp\p(A(£ ■ 
£,')&' + B(£ • £'))] is true for a' 6 {—1,1}. For instance, for the expression ( |T8| ) of sequential 
dynamics this leads to: 



lim 



where 



1 



log R 



seq 



lim 

N- 



1 



hi)] 



h 



i+l 



i+l 



log 



cosh[/?(J? i+1 - ^)] 
cosh[/J( J? i+1 + /*)] 



(with /ii = im£i) and is in complete agreement with the free energy as found from (16,32,38). 

Finally, for the special case = Jg = J?' = and p = 1 (short-range bond disorder and 
absence of long-range interactions) our model reduces to the classical short-range random-bond 

to acquire, at least in 



Ising model [12], in which we expect the integrated density Poo(k) 
certain parameter regions, the form of the highly non-analytic Devil's Staircase [13, |lif| . In this 
special case the density (40) reduces (at saddle points {irho = —8,im = 0}) to 



Pi+i(k,\) 




e /3J s X _ fc e -/3J s \ 



M+Pi 



,2139 



e -/3J s X _ k e PJ 8 \ 



where A = ±1 represents bond-disorder. One can now prove by induction that if the identity 
Pi(k, 1) = Pi(k, —1) is true then also Pi + \{k, 1) = Pi + i(k, —1), so that (assuming ergodicity and 
uniqueness of the stationary density) the above expression reduces to a single recursive equation: 



P+Ak)- l -{p (f*> ^ ~ ( 



p ( 20e ke-V> - 




— keP Ja 



which is recognised as equation 10 of Ref. [12j , upon a simple re-definition of our stochastic 
variables: {k n = e~ 2/3m ° R n ^/ R n ^} — * {e 2/3im ° R n ^ / R n ^} in fl26]). In the present benchmark 
case we have also verified the identity found in |]l6[ relating short-range random-field models 
between sequential and parallel dynamics, namely 

V-seq frMfc ; 0) , Ce, e) = Vr (k ; t'Z, o) 

(which is the key identity to prove that in the thermodynamic limit sequential and parallel 
random-field models lead to the same physical states). Here, VW}(- • •) and V , par(- • •) correspond 
to the functions defined in (|30j and (|3l|). 
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6 Theory Vs Simulations 



In order to test our results further, we have performed extensive simulation experiments of the 
process (|l]), for model (|4|). In all cases the initial state is prepared randomly, with non-zero 
correlation only with pattern Our simulation results for the model which gives the phase 

diagram of figure ^ (p = 2 and jj 1 ^ = Jg = 0) are shown in figure |?j (upper row) where we 
draw the equilibrium value of the recall overlap m\{t —* oo) as a function of the initial state 
mi{t = 0). We have performed our experiments for three different regions of the phase diagram: 
region B (one m > stable state), region C (two m > stable states) and region F (three 
m > stable states). In regions B and C the simulation experiments verify the appearance 
and location of multiple ergodic sectors; to compare with the theoretical results, see also the 
free energy graphs in the middle and lower pictures of figure 0. In region F the simulation 
experiments show that the system can enter only two possible domains of attraction (excluding 
thus the theoretically predicted state m ~ 0.32). This is due to (i) the system's finite size 
(N = 1,000), in combination with (ii) the (relatively) small energy barrier separating the two 
physical states m a ~ 0.32 and ~ 0.68 (this allows the system to move from state m a to mj 
with a non- negligible probability). Our restriction to system size N = 1,000 is prompted by 
the extremely long equilibration times (of the order of O(10 6 ) flips/spin). This, in turn, is due 
to domain formation: large clusters of neurons tend to freeze in specific configurations. As a 
consequence, in order for neurons to flip, the entire domain has to flip. In figure ^ (left graph) we 
show the value of the condensed overlap as a function of time in region F. We see that, starting 
from an initial state mi(0) ~ 0.09, the system gradually approaches the theoretically predicted 
locally stable state, where it indeed stays for a period of ~ 4 • 10 5 flips/spin. Due to finite 
size effects, however, this state is thermodynamically unstable. A sudden transition to a new 
meta-stable state is then observed, generated by the flipping of entire domains. Equilibrium is 
reached in these simulations at about 2 • 10 6 flips/spin, where a second and final jump transition 
takes place. In the right graph of figure |8] we show a simulation experiment carried out in region 
C, starting from initial conditions TOi(0) ~ 0.08. Here equilibrium is reached after about 10 6 
flips/spin, and due to a (relatively) high energy barrier separating the two m > physical states 
(see energy graphs in regions C, left column of figure 0) there is no domain-related transition. 
In all our experiments the value of the non-selected pattern overlap m2(t) is found to remain 
zero (open diamond points), which justifies a posteriori our pure state ansatz. 



7 Discussion 



In this paper we have presented an exact equilibrium solution for a specific class of spatially 
structured Ising spin (attractor) neural network models, in which there is competition induced 
by the presence of two qualitatively different types of synaptic interactions: those operating only 
between nearest neighbours in a ID chain (short-range), and those operating between any pair 
of neurons (long-range). The values taken by the interactions present are given by Hebbian- 
type rules, as in the more familiar mean-field attractor networks. We have solved these models 
by using a combination of mean- and random-field techniques for both sequential and parallel 
dynamics. As in the standard ID RFIM-type models our expressions for the disorder-averaged 
free energy per neuron take the form of integrals over the distribution of a random variable, 
which represents the ratio of conditioned partition functions. This distribution can be 
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Figure 7: Upper row: sequential dynamics simulation results of the dynamical process (|l|), with model 
(|J), for a system with p = 2 patterns. These were carried out in three different regions C,F and B of the 
phase diagram of figure ||. System size: N = 1,000. Initial conditions are random, subject to prescribed 
correlations with pattern We show the equilibrium state m(t — > oo) of the 'pure state' overlap 

mi (full circles), as well as the overlap rri2 (open diamonds), as functions of the initial state m(t = 0). 
Finite size effects are of the order 0(JV -1 / 2 ) 0.03. Middle and lower rows: free energy per neuron 
</> seq (m, im), after elimination of the conjugate order parameter m via d m <p seq (m,im) = (middle row), 
and similarly after elimination of rh via d^(j) se q\m,im) = (lower row). Left column: (3J\ — 14 and 
(3J^ — —3.5 (region C of the phase diagram of figure^), middle column: (3J^ — 18.5 and /3ji 2 ' = —4 
(region F), and right column: [3J^ = 8 and [3J^ = —3.5 (region B). For all graphs J, 1 ' — = 0. 
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Figure 8: Simulation results for regions F (left picture) and C (right picture) of the phase diagram of 
figure ||. We show the evolution of the 'pure state' overlap order parameter as a function of time. In 
region F (left) the theory predicts a locally stable state at m w 0.32, which, due to finite size effects, 
appears here only as a meta-stable state. Two prominent jump transitions occur, until finally equilibrium 
is reached, at m w 0.68 (the jumps indicate domain-flipping) . In region C (right) full equilibration still 
requires simulation times of the order of O(10 6 ) flips/spin, after which the result is in true agreement 
with the theory 



evaluated numerically without much effort, and the key macroscopic observables then follow via 
simple differentiation. 

We found that there are regions in parameter space where information processing between 
the two types of synaptic interactions can induce phenomena which are quite novel in the arena of 
associative memory models, such as the appearance of multiple locally stable states, and of first- 
order transitions between them, even for finite p and upon making the 'pure state' ansatz. These 
peculiarities come to life particularly in regions where short- and long-range synapses compete 
most strongly, for instance, where one has Hebbian long-range interactions in combination with 
anti-Hebbian short-range ones, and they become more evident when increasing the number of 
stored patterns. Particularly in the upper left quadrant of parameter space { Je > 0, J s < 0} 
one observes the appearance of an increasing number of dynamic transition lines (first- and 
second-order ones). This feature is in sharp contrast with the conventional (infinite-range) 
Hopfield-like networks Q, where for finite p the 'pure state' ansatz automatically renders the 
remaining order parameter independent of the number of patterns stored. Phenomena such as 
simultaneous existence of multiple locally stable states (in which the quality of pattern recall 
depends crucially on initial conditions) can play a potentially useful role: poor cue signals 
can no longer evoke pattern recall. We also found that parallel dynamics transition lines in 
parameter space are exact reflection in the origin of those in sequential dynamics and that the 
relevant macroscopic observables can be obtained from those of sequential dynamics via simple 
transformations. Simulation experiments also show that the dynamics of the model are highly 
non-trivial, with plateaus and jump discontinuities, caused by complex domain formation and 
domain interaction, which would justify a study in itself. 

In a similar fashion one could now also study more complicated systems, where (in addition 
to the long-range synapses) the short-range synapses reach beyond nearest neighbours. Such 
models can still be solved using the techniques employed here. A different type of generalisation 
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would be to allow for a type of competition between synapses which would correspond to having 
stored patterns with different (pattern dependent) embedding strengths, as in |19| ]. All these 
will be subjects of a future work. 
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